Conveying Spatial Information Using Vision and Natural Language

Language Alicia Abella John R. Kender Department of Computer Science Columbia University New York, NY 10027 Abstract Technology has made it possible to gather a great deal of data in the form of images, and at the same time it has created a demand for extracting what is conveyed in these images. To extract what is in these images requires vision, and to convey what is extracted requires natural language. This paper examines the problem of how computer vision and natural language processing can be used to address the problem of object localization in a 2D image. The ultimate goal is a system capable of generating descriptions that relate the spatial arrangement of the objects through the use of spatial prepositions. Our previous work described a computational model of preposition semantics and a method for handling some of the ambiguities associated with natural language. In this paper we extend that work by introducing the theoretical methodologies for generating locative expressions that represent the spatial relationship of objects in an image. Introduction The integration of vision and natural language has recently emerged as an area of crucial importance to tasks that involve communicating visual information. Figure 1 illustrates the current state of a airs. There is a signi cant amount of theoretical results and systems built in the areas of computer vision and natural language processing. The area formed by their overlap, however, contains comparably few theoretical results, and even fewer successful implementations that utilize vision and natural language. This paper presents a semantic representation of spatial prepositions based on an image's visual properties for the purpose of describing the spatial relationship of objects in an image. This semantic representation relies on object properties extracted from the image using standard vision techniques, see e.g.[Horn, 1990]. These properties include an object's area, center of mass, and elongation. An early attempt at de ning a model for language understanding and developing a system to test it is il?